A cluster file system allows multiple client devices to share access to files over a network. One well-known cluster file system is the Lustre file system. Lustre is a Linux-based high performance cluster file system utilized for computer clusters ranging in size from small workgroup clusters to large-scale, multi-site clusters. Lustre can readily scale to support tens of thousands of clients, petabytes of storage capacity, and hundreds of gigabytes per second of aggregate input-output (IO) throughput. Due to its high performance and scalability, Lustre is utilized in many supercomputers, as well as other complex computing environments, including large enterprise data centers.
In many parallel computing applications, including cluster file systems, a group of distributed processes must often write data to a shared file. When multiple processes attempt to write data to a shared file concurrently, however, the performance of the parallel storage system will be impaired. Serialization can cause significant performance degradation as the parallel processes must remain idle while they wait for one another. Serialization is incurred when the parallel file system locks a shared file in order to maintain the consistency of the shared file.
A number of techniques have been proposed or suggested to organize the data streams when multiple processes simultaneously save data to a shared file. For example, each process can create a single file across a set of different directories and then sequentially write a large amount of data to the single file. In a further implementation, a single process (often referred to as a “leader”) can create a shared file, and then all the processes write to the shared file in segments that are aligned with block boundaries within the parallel file system.
A need therefore exists for improved techniques for writing data from a group of distributed processes to a shared file in a cluster file system.